Estimating the efficiency gain of covariate-adjusted analyses in future clinical trials using external data

We present a framework for using existing external data to identify and estimate the relative efficiency of a covariate-adjusted estimator compared to an unadjusted estimator in a future randomized trial. Under conditions, these relative efficiencies approximate the ratio of sample sizes needed to achieve a desired power. We develop semiparametrically efficient estimators of the relative efficiencies for several treatment effect estimands of interest with either fully or partially observed outcomes, allowing for the application of flexible statistical learning tools to estimate the nuisance functions. We propose an analytic Wald-type confidence interval and a double bootstrap scheme for statistical inference. We demonstrate the performance of the proposed methods through simulation studies and apply these methods to estimate the efficiency gain of covariate adjustment in Covid-19 therapeutic trials.

The effect of vaccine on COVID-19 spread by function-on-scalar regression model: a case study of Africa.

Aim: This paper aimed to study the effect of the vaccine on the reproduction rate of coronavirus in Africa from January 2021 to November 2021. Subject and methods: Functional data analysis (FDA), a relatively new area in statistics, can describe, analyze, and predict data collected over time, space, or other continuum measures in many countries every day and is increasingly common across scientific domains. For our data, the first step of functional data is smoothing. We used the B-spline method to smooth our data. Then, we apply the function-on-scalar and Bayes function-on-scalar models to fit our data. Results: Our results indicate a statistically significant relationship between the vaccine and the rate of virus reproduction and spread. When the vaccination rate falls, the reproduction rate also decreases. Furthermore, we found that the effect of latitude and the region on the reproduction rate depends on the region. We discovered that in Middle Africa, from the beginning of the year until the end of the summer, the impact is negative, implying that the virus spread due to a decrease in the vaccination rates. Conclusion: The study found that vaccination rates significantly impact the virus's reproduction rate.

Impact of the Covid-19 pandemic on the performance of machine learning algorithms for predicting perioperative mortality.

BACKGROUND: Machine-learning models are susceptible to external influences which can result in performance deterioration. The aim of our study was to elucidate the impact of a sudden shift in covariates, like the one caused by the Covid-19 pandemic, on model performance. METHODS: After ethical approval and registration in Clinical Trials (NCT04092933, initial release 17/09/2019), we developed different models for the prediction of perioperative mortality based on preoperative data: one for the pre-pandemic data period until March 2020, one including data before the pandemic and from the first wave until May 2020, and one that covers the complete period before and during the pandemic until October 2021. We applied XGBoost as well as a Deep Learning neural network (DL). Performance metrics of each model during the different pandemic phases were determined, and XGBoost models were analysed for changes in feature importance. RESULTS: XGBoost and DL provided similar performance on the pre-pandemic data with respect to area under receiver operating characteristic (AUROC, 0.951 vs. 0.942) and area under precision-recall curve (AUPR, 0.144 vs. 0.187). Validation in patient cohorts of the different pandemic waves showed high fluctuations in performance from both AUROC and AUPR for DL, whereas the XGBoost models seemed more stable. Change in variable frequencies with onset of the pandemic were visible in age, ASA score, and the higher proportion of emergency operations, among others. Age consistently showed the highest information gain. Models based on pre-pandemic data performed worse during the first pandemic wave (AUROC 0.914 for XGBoost and DL) whereas models augmented with data from the first wave lacked performance after the first wave (AUROC 0.907 for XGBoost and 0.747 for DL). The deterioration was also visible in AUPR, which worsened by over 50% in both XGBoost and DL in the first phase after re-training. CONCLUSIONS: A sudden shift in data impacts model performance. Re-training the model with updated data may cause degradation in predictive accuracy if the changes are only transient. Too early re-training should therefore be avoided, and close model surveillance is necessary.

How well do covariates perform when adjusting for sampling bias in online COVID-19 research? Insights from multiverse analyses.

COVID-19 research has relied heavily on convenience-based samples, which-though often necessary-are susceptible to important sampling biases. We begin with a theoretical overview and introduction to the dynamics that underlie sampling bias. We then empirically examine sampling bias in online COVID-19 surveys and evaluate the degree to which common statistical adjustments for demographic covariates successfully attenuate such bias. This registered study analysed responses to identical questions from three convenience and three largely representative samples (total N = 13,731) collected online in Canada within the International COVID-19 Awareness and Responses Evaluation Study ( www.icarestudy.com ). We compared samples on 11 behavioural and psychological outcomes (e.g., adherence to COVID-19 prevention measures, vaccine intentions) across three time points and employed multiverse-style analyses to examine how 512 combinations of demographic covariates (e.g., sex, age, education, income, ethnicity) impacted sampling discrepancies on these outcomes. Significant discrepancies emerged between samples on 73% of outcomes. Participants in the convenience samples held more positive thoughts towards and engaged in more COVID-19 prevention behaviours. Covariates attenuated sampling differences in only 55% of cases and increased differences in 45%. No covariate performed reliably well. Our results suggest that online convenience samples may display more positive dispositions towards COVID-19 prevention behaviours being studied than would samples drawn using more representative means. Adjusting results for demographic covariates frequently increased rather than decreased bias, suggesting that researchers should be cautious when interpreting adjusted findings. Using multiverse-style analyses as extended sensitivity analyses is recommended.

Optimising precision and power by machine learning in randomised trials with ordinal and time-to-event outcomes with an application to COVID-19.

The rapid finding of effective therapeutics requires efficient use of available resources in clinical trials. Covariate adjustment can yield statistical estimates with improved precision, resulting in a reduction in the number of participants required to draw futility or efficacy conclusions. We focus on time-to-event and ordinal outcomes. When more than a few baseline covariates are available, a key question for covariate adjustment in randomised studies is how to fit a model relating the outcome and the baseline covariates to maximise precision. We present a novel theoretical result establishing conditions for asymptotic normality of a variety of covariate-adjusted estimators that rely on machine learning (e.g., ℓ 1 -regularisation, Random Forests, XGBoost, and Multivariate Adaptive Regression Splines [MARS]), under the assumption that outcome data are missing completely at random. We further present a consistent estimator of the asymptotic variance. Importantly, the conditions do not require the machine learning methods to converge to the true outcome distribution conditional on baseline variables, as long as they converge to some (possibly incorrect) limit. We conducted a simulation study to evaluate the performance of the aforementioned prediction methods in COVID-19 trials. Our simulation is based on resampling longitudinal data from over 1500 patients hospitalised with COVID-19 at Weill Cornell Medicine New York Presbyterian Hospital. We found that using ℓ 1 -regularisation led to estimators and corresponding hypothesis tests that control type 1 error and are more precise than an unadjusted estimator across all sample sizes tested. We also show that when covariates are not prognostic of the outcome, ℓ 1 -regularisation remains as precise as the unadjusted estimator, even at small sample sizes ( n = 100 ). We give an R package adjrct that performs model-robust covariate adjustment for ordinal and time-to-event outcomes.

An alternative to Cox's regression for multiple survival curves comparison: A random forest-based approach using covariate structure

There are several established methods for comparing more than two survival curves, namely the scale-rank test or Cox's proportional hazard model. However, when their statistical assumptions are not met, their results' validity is affected. In this study, we address the mentioned issue and propose a new statistical approach on how to compare more than two survival curves using a random forest algorithm, which is practically assumption-free. The repetitive generating of many decision trees covered by one random forest model enables to calculate of a proportion of trees with sufficient complexity classifying into all groups (depicted by their survival curves), which is the p-value estimate as an analogy of the classical Wald's t-test output of the Cox's regression. Furthermore, a level of the pruning of decision trees the random forest model is built with, can modify both the robustness and statistical power of the random forest alternative. The discussed results are confirmed using COVID-19 survival data with varying the tree pruning level. The introduced method for survival curves comparison, based on random forest algorithm, seems to be a valid alternative to Cox's regression;however, it has no statistical assumptions and tends to reach higher statistical power. © 2021 IEEE

Interpretable Temporal Attention Network for COVID-19 forecasting.

The worldwide outbreak of coronavirus disease 2019 (COVID-19) has triggered an unprecedented global health and economic crisis. Early and accurate forecasts of COVID-19 and evaluation of government interventions are crucial for governments to take appropriate interventions to contain the spread of COVID-19. In this work, we propose the Interpretable Temporal Attention Network (ITANet) for COVID-19 forecasting and inferring the importance of government interventions. The proposed model is with an encoder-decoder architecture and employs long short-term memory (LSTM) for temporal feature extraction and multi-head attention for long-term dependency caption. The model simultaneously takes historical information, a priori known future information, and pseudo future information into consideration, where the pseudo future information is learned with the covariate forecasting network (CFN) and multi-task learning (MTL). In addition, we also propose the degraded teacher forcing (DTF) method to train the model efficiently. Compared with other models, the ITANet is more effective in the forecasting of COVID-19 new confirmed cases. The importance of government interventions against COVID-19 is further inferred by the Temporal Covariate Interpreter (TCI) of the model.

A comparison of multiple neighborhood matrix specifications for spatio-temporal model fitting: a case study on COVID-19 data.

Establishing proper neighbor relations between a set of spatial units under analysis is essential when carrying out a spatial or spatio-temporal analysis. However, it is usual that researchers choose some of the most typical (and simple) neighborhood structures, such as the first-order contiguity matrix, without exploring other options. In this paper, we compare the performance of different neighborhood matrices in the context of modeling the weekly relative risk of COVID-19 over small areas located in or near Valencia, Spain. Specifically, we construct contiguity-based, distance-based, covariate-based (considering mobility flows and sociodemographic characteristics), and hybrid neighborhood matrices. We evaluate the goodness of fit, the overall predictive quality, the ability to detect high-risk spatio-temporal units, the capability to capture the spatio-temporal autocorrelation in the data, and the goodness of smoothing for a set of spatio-temporal models based on each of the neighborhood matrices. The results show that contiguity-based matrices, some of the distance-based matrices, and those based on sociodemographic characteristics perform better than the matrices based on k-nearest neighbors and those involving mobility flows. In addition, we test the linear combination of some of the constructed neighborhood matrices and the reweighting of these matrices after eliminating weak neighbor relations, without any model improvement.

Estimation of the odds ratio in a proportional odds model with censored time-lagged outcome in a randomized clinical trial.

In many randomized clinical trials of therapeutics for COVID-19, the primary outcome is an ordinal categorical variable, and interest focuses on the odds ratio (OR; active agent vs control) under the assumption of a proportional odds model. Although at the final analysis the outcome will be determined for all subjects, at an interim analysis, the status of some participants may not yet be determined, for example, because ascertainment of the outcome may not be possible until some prespecified follow-up time. Accordingly, the outcome from these subjects can be viewed as censored. A valid interim analysis can be based on data only from those subjects with full follow-up; however, this approach is inefficient, as it does not exploit additional information that may be available on those for whom the outcome is not yet available at the time of the interim analysis. Appealing to the theory of semiparametrics, we propose an estimator for the OR in a proportional odds model with censored, time-lagged categorical outcome that incorporates additional baseline and time-dependent covariate information and demonstrate that it can result in considerable gains in efficiency relative to simpler approaches. A byproduct of the approach is a covariate-adjusted estimator for the OR based on the full data that would be available at a final analysis.

The Problem of Fairness in Synthetic Healthcare Data.

Access to healthcare data such as electronic health records (EHR) is often restricted by laws established to protect patient privacy. These restrictions hinder the reproducibility of existing results based on private healthcare data and also limit new research. Synthetically-generated healthcare data solve this problem by preserving privacy and enabling researchers and policymakers to drive decisions and methods based on realistic data. Healthcare data can include information about multiple in- and out- patient visits of patients, making it a time-series dataset which is often influenced by protected attributes like age, gender, race etc. The COVID-19 pandemic has exacerbated health inequities, with certain subgroups experiencing poorer outcomes and less access to healthcare. To combat these inequities, synthetic data must "fairly" represent diverse minority subgroups such that the conclusions drawn on synthetic data are correct and the results can be generalized to real data. In this article, we develop two fairness metrics for synthetic data, and analyze all subgroups defined by protected attributes to analyze the bias in three published synthetic research datasets. These covariate-level disparity metrics revealed that synthetic data may not be representative at the univariate and multivariate subgroup-levels and thus, fairness should be addressed when developing data generation methods. We discuss the need for measuring fairness in synthetic healthcare data to enable the development of robust machine learning models to create more equitable synthetic healthcare datasets.

Hawkes process modeling of COVID-19 with mobility leading indicators and spatial covariates.

Hawkes processes are used in statistical modeling for event clustering and causal inference, while they also can be viewed as stochastic versions of popular compartmental models used in epidemiology. Here we show how to develop accurate models of COVID-19 transmission using Hawkes processes with spatial-temporal covariates. We model the conditional intensity of new COVID-19 cases and deaths in the U.S. at the county level, estimating the dynamic reproduction number of the virus within an EM algorithm through a regression on Google mobility indices and demographic covariates in the maximization step. We validate the approach on both short-term and long-term forecasting tasks, showing that the Hawkes process outperforms several models currently used to track the pandemic, including an ensemble approach and an SEIR-variant. We also investigate which covariates and mobility indices are most important for building forecasts of COVID-19 in the U.S.

Discussion on "Improving precision and power in randomized trials for COVID-19 treatments using covariate adjustment for binary, ordinal, and time-to-event outcomes".

Benkeser et al. present a very informative paper evaluating the efficiency gains of covariate adjustment in settings with binary, ordinal, and time-to-event outcomes. The adjustment method focuses on estimating the marginal treatment effect averaged over the covariate distribution in both arms combined. The authors show that covariate adjustment can achieve power gains that could find answers more quickly. The suggested approach is an important weapon in the armamentarium against epidemics like COVID-19. I recommend evaluating the procedure against more traditional approaches for conditional analyses (e.g., logistic regression) and against blinded methods of building prediction models followed by randomization-based inference.

A randomized and comparative study to assess safety and efficacy of supplemental treatment of a herbal formulation - Aayudh Advance comprising essential oils in patients with corona virus 2019 (COVID-19).

OBJECTIVE: The purpose of this study was to examine the effect of herbal formulation - Aayudh Advance on viral load as well as recovery duration in mild symptomatic patients diagnosed with Corona Virus Disease 2019 (COVID-19). It also aimed to study the effect of Herbal formulation - Aayudh Advance in terms of clinical improvement of various sign and symptoms in mild symptomatic COVID-19 patients. METHOD: Once the patient suffice the requirement of inclusion, exclusion criteria of the study than as per the method of 'Covariate Adaptive Randomization' technique, patient was assigned in either Aayudh Advance arm (Test arm) or Control Arm. Here standard of Care treatment was given to all patients of both the arms. Treatment was given for the period of 14 days or till patient turned COVID-19 negative, which ever was earlier. Clinical signs and symptoms viz. body temperature, SpO 2, Scoring of Cough & Scoring of Shortness of breath were recorded on all 5 Clinical visits along with biochemical testing like RT-PCR (with CT value of E gene and RDRP gene), serum ferritin, CRP and NLR observed on weekly Visit. RESULT: Total 74 patients were enrolled in the present study. Out of which 60 patients (30 patients in each group) have completed study as per the protocol, whereas 14 patients have voluntarily withdrawn from the study due to getting early discharge from the hospital. All patients in Aayudh Advance treatment group recovered (100%) after 14 days. This observed recovery was 15.38% more as compared to Standard of Care treatment alone. Further, there was statistically significant reduction (p < 0.05) in viral load as indicated by significant increase in CT value of E-gene and RDRP gene. Further, no patients reported any Adverse Reaction as well as no drug to drug interaction was observed with supplemental treatment with Aayudh Advance. CONCLUSION: The Aayudh Advance was found safe as well as more effective in terms of reduction of viral load. % recovery was more in Treatment arm as compared to Control arm in mild symptomatic COVID-19 patients.

Improving precision and power in randomized trials for COVID-19 treatments using covariate adjustment, for binary, ordinal, and time-to-event outcomes.

Time is of the essence in evaluating potential drugs and biologics for the treatment and prevention of COVID-19. There are currently 876 randomized clinical trials (phase 2 and 3) of treatments for COVID-19 registered on clinicaltrials.gov. Covariate adjustment is a statistical analysis method with potential to improve precision and reduce the required sample size for a substantial number of these trials. Though covariate adjustment is recommended by the U.S. Food and Drug Administration and the European Medicines Agency, it is underutilized, especially for the types of outcomes (binary, ordinal, and time-to-event) that are common in COVID-19 trials. To demonstrate the potential value added by covariate adjustment in this context, we simulated two-arm, randomized trials comparing a hypothetical COVID-19 treatment versus standard of care, where the primary outcome is binary, ordinal, or time-to-event. Our simulated distributions are derived from two sources: longitudinal data on over 500 patients hospitalized at Weill Cornell Medicine New York Presbyterian Hospital and a Centers for Disease Control and Prevention preliminary description of 2449 cases. In simulated trials with sample sizes ranging from 100 to 1000 participants, we found substantial precision gains from using covariate adjustment-equivalent to 4-18% reductions in the required sample size to achieve a desired power. This was the case for a variety of estimands (targets of inference). From these simulations, we conclude that covariate adjustment is a low-risk, high-reward approach to streamlining COVID-19 treatment trials. We provide an R package and practical recommendations for implementation.